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Watermark Encoding and Decoding in Imaging Devices and 

Imaging Device Interfaces 

Related Application Data 

This patent application is related to co-pending U.S. patent application 
5 09/428,359, filed October 28, 1999, which is a division of co-pending application 

09/185,380, filed November 3, 1998. A PCT counterpart of the '380 application has been 
published as WIPO publication WO0026749. This patent application is also related to- 
pending application 09/465,418, filed December 16, 1999, which claims priority benefit 
f to provisional application 60/1 12,955, filed December 18, 1998. 
f 0 The subject matter of the present application is related to that disclosed in US 

I Patent 5,862,260, and in co-pending application 09/503,881, filed February 14, 2000; 
' which are hereby incorporated by reference. 

Technical Field 

J 5 The invention relates to digital watermarking technology, and specifically relates 

f to incorporating such technology in imaging devices such as scanners and printers and 
interfaces for these devices. 

Background and Summary 

Digital watermarking is a process for modifying physical or electronic media to 
20 embed a machine-readable code into the media. The media may be modified such that 
the embedded code is imperceptible or nearly imperceptible to the user, yet may be 
detected through an automated detection process. Most commonly, digital watermarking 
is applied to media signals such as images, audio signals, and video signals. However, it 
may also be applied to other types of media objects, including documents (e.g., through 
25 line, word or character shifting), software, multi-dimensional graphics models, and 
surface textures of objects. 


t 
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Digital watermarking systems typically have two primary components: an 
encoder that embeds the watermark in a host media signal, and a decoder that detects and 
reads the embedded watermark from a signal suspected of containing a watermark (a 
suspect signal). The encoder embeds a watermark by altering the host media signal. The 
5 reading component analyzes a suspect signal to detect whether a watermark is present. In 
applications where the watermark encodes information, the reader extracts this 
information from the detected watermark. 

Several particular watermarking techniques have been developed. The reader is 
presumed to be familiar with the literature in this field. Particular techniques for 
1 0 embedding and detecting imperceptible watermarks in media signals are detailed in the 
! assignee's co-pending application serial number 09/503,881 and US Patent 5,862,260, 
% which are hereby incorporated by reference. 

: The invention relates to watermark encoding and decoding in imaging devices, 

such as printers and scanners, and imaging device interfaces. One aspect of the invention 

1 5 is a method and system for image watermark decoding in a print process pipeline. The 
method performs watermark decoding on an image as it is being transferred from one 

I stage of a printing process to another. In particular, in one implementation, a streaming 
mode watermark decoder operates on sequential portions of the image in a sequential 

I stream passing from one stage of a print process to another. The streaming mode decoder 

20 may be incorporated in a printer driver in a computer connected to a printer peripheral or 
within a printer device. 

This type of watermark decoder can be incorporated into a printer driver or printer 
device for a variety of applications. One application is counterfeit deterrence. 
Specifically, the decoder can be used to detect a watermark in a high value document 

25 (e.g., bank note, check, authentication label, ticket, identity document, etc.) as it is being 
printed and inhibit the printing process so that printer output is incomplete. Another 
application is to associate the image being printed with metadata and to render that 
metadata during the print process. The watermark includes metadata or a reference to 
metadata in an external database (either within the printer system or in a remote database 


i 
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via a network connection, e.g., on the Internet). Rendering the metadata may include 
displaying information and/or executing some program or hardware function associated 
with the image being printed, such as fetching and rendering a related web page 
referenced via the watermark message. 
5 Another aspect of the invention is a method and system for image watermark 

encoding in a print process pipeline. The method performs watermark encoding on an 
image as it is being transferred from one stage of a printing process to another. In 
particular, in one implementation, a streaming mode watermark encoder operates on 
sequential portions of the image in a sequential stream passing from one stage to another. 

1 0 This type of encoder can be incorporated into a printer driver or printer device for a 

1 variety of applications. 

I One application is to encode user information as a document is being printed. 

: This user information may be used for counterfeit deterrence by embedding tracer 

information in the document that will help identify the maker of the counterfeit 
15 document. Another application is to associate other forms of metadata about the 

document as it is being printed by embedding the metadata or a reference to the metadata 
1 in a watermark. 

, Yet another aspect of the invention is a method and system for image watermark 

\ decoding in a scanner pipeline. The method performs watermark decoding on an image 
20 as it is being transferred from one stage of a scanning process to another. In particular, in 
one implementation, a streaming mode watermark decoder operates on sequential 
portions of the image in a sequential stream passing from one stage to another. This type 
of decoding can be used for a variety of applications, including those mentioned 
previously for printer drivers and devices. In particular, it can be used to discontinue the 
25 unauthorized image scan of a high value document, such as a bank note, identify 

document, ticket, check, etc. Further, information decoded from the watermark may 
include metadata or a reference to metadata in an external database. Upon decoding this 
information, the watermark decoder fetches and renders the metadata. 
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Yet another aspect of the invention is a method and system for image watermark 
encoding in a scanner pipeline. The method performs watermark encoding on an image 
as it is being transferred from one stage of a scanning process to another. In particular, in 
one implementation, a streaming mode watermark encoder operates on sequential 
5 portions of the image in a sequential stream passing from one stage to another. This type 
of decoding can be used for a variety of applications, including those mentioned 
previously for printer drivers and devices. One application is to encode user information 
in an image before it is transmitted to another process or application. For example, in fax 
machines, the encoder may be used to embed information about the sender or receiver of 

10 a fax image in the image before transmitting the image over a telephone connection. 

* By incorporating watermark functions in imaging devices or their drivers, these 

1 functions are inherently available for many software processes or hardware devices that 

i communicate with the imaging devices. 

; Further features will become apparent with reference to the following detailed 

A 5 description and accompanying drawings. 


Brief Description of the Drawings 

: Fig. 1 is a diagram of a streaming mode watermark decoder. 

Fig. 2 is a diagram of another streaming mode watermark decoder. 
20 Fig. 3 is a diagram of a streaming mode watermark encoder. 


Detailed Description 

There are a number of applications where it is advantageous to incorporate image 
watermark encoding and decoding functions into imaging devices, such as scanners and 
25 printers, and interfaces of these devices, such as software drivers. One application is 
counterfeit deterrence where watermarks are used to deter reproduction of high value 
documents such as bank notes, identity documents, packaging, labels, certificates of 
authentication, tickets, etc. Another application is to use information conveyed in a 
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watermark to control use, transfer or rendering of a watermarked image. Yet another 
application is to associate metadata with an image. 

To incorporate these watermarking applications in imaging devices and their 
interfaces, it is beneficial to be able to perform watermarking operations in a streaming 
5 mode on portions of an image as it is being scanned or printed. In this context, streaming 
mode refers to a type of watermark function that operates on sequential portions of an 
image as these portions are being transferred from one process or device to another. It is 
typical, for example, for software drivers to pass sequential portions of an image to a 
printer for printing. Similarly, it is typical for software drivers of a scanner to receive 
10 sequential portions of an image from a scanning device. In many cases, it is necessary or 
: beneficial for watermark functions to operate on a sequential portion of an image without 
1 re-accessing image data that has previously passed and to complete operation on one 
I portion before beginning the same operation on the next portion in a sequential stream. 
* Fig. 1 is a diagram of a streaming mode watermark decoder. The streaming mode 

11 5 watermark decoder processes incoming portions of an image to detect the presence of a 
% watermark, and read a message encoded in the watermark. The detector intercepts and 
f buffers portions of image data as it is being transferred. The buffer holds a most recently 
i= received portion of the image. It analyzes blocks of image data in the buffer to assess the 
\ likelihood that they include a known type of watermark. For candidate blocks that it 
20 identifies, the detector proceeds to execute watermark detection and read operations. 

Looking at Fig. 1 in more detail, the decoder operates on incoming bands (100) of 
image data. A band is a set of contiguous scan lines of pixels in the image. The decoder 
temporarily stores the N most recent bands of image data in a First In-First Out (FIFO) 
buffer (102). The decoder selects the number N based on the height of the bands so that 
25 the FIFO holds a sequential portion of the image that is at least as high as a block of 
image data. A block is a unit of image data upon which the decoder performs decoding 
operations. The size and shape of a band and block may vary with the implementation, 
depending on such factors as the watermark decoding algorithm and watermark signal 
protocol. In a previous embedding process on the image, a watermark encoder replicates 
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instances of a watermark signal throughout the image, and the block size is sufficiently 
large to enable the decoder to detect a watermark instance, and possibly recover an entire 
watermark message from that instance. 

The format of the image data in the incoming band and FIFO vary with the 
5 implementation, depending on a number of considerations. One consideration is memory 
usage of the FIFO buffer. Another consideration is processing time. Another 
consideration is the type of image data needed to perform a decoding operation (e.g., full 
color information, luminance information, a binary bitmap such as a half-tone image, 
etc.). To save memory and processing time, the decoder may operate on down-sampled 
10 image data. In addition, rather than requiring color data per pixel, such as RGB or YUV 
I triplets per pixel, the decoder may operate on single color channel, such as a luminance 
1 value per pixel or on half-tone, binary bitmap information. In some cases, as noted 
I below, additional image data, such as color information is preferable over a single color 
* channel, like luminance, because it may be used to improve the accuracy of a watermark 
fl 5 detection and message recovery. 

, The decoder includes a cache manager (1 04) that maintains a set of image blocks 

I from the FIFO buffer in a block cache (1 06). The block cache stores image data and 
i block location information for each block. The decoder performs detection operations on 
; the set of image blocks in this cache. To fill the cache, the cache manager analyzes 
20 image data in the FIFO and selects a set of K blocks that are likely candidates for having 
a recoverable watermark signal. 

The criteria used to select these candidate blocks depends on the watermark 
embedding function and known attributes of watermarked images. The cache manager 
selects blocks having statistical features (signal activity, edges, colors) that make them 
25 likely candidates for having a detectable and recoverable watermark signal. In one 

implementation, a watermark encoder embeds an auxiliary message into a host signal by 
adding a perceptually adapted, pseudorandom pattern into a particular color channel of 
certain types of documents. This pattern is perceptually adapted such that signal intensity 
of the pattern increases in noisy areas of the image. The noisy blocks provide good 
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candidates for hiding a watermark in the encoder, and for detecting and recovering a 
watermark in the decoder. As such, the cache manager ranks blocks based on a measure 
of watermark hiding and detectability characteristics. Specific measures of these 
characteristics include signal activity or signal energy and the quantity of edges in a 
5 candidate block. 

The cache manager may also use known attributes of the types of documents 
containing watermarks to analyze the blocks. For example, some types of watermarks 
may be known to have a particular color composition. By looking at color data, the cache 
manager can rank blocks based at least in part on whether the colors match colors known 
10 to be in certain watermarked documents. 

When adequate new blocks are available, the cache manager 104 instructs a 
I detector module 1 08 to perform watermark detection functions on the K candidate blocks 
| in the cache. The specific criteria for triggering a detection operation vary with the 
* implementation. Some examples of the criteria include: 1) when the cache manager has 
5 identified a new candidate block or blocks; 2) when N new bands are loaded into the 
FIFO; 3) criteria based on attributes of watermarked images being sought (dimensions of 
I watermarked image, colors of watermarked image, evidence of visible patterns detected 
[ in image, etc.). 

I Once triggered, detection stages in the detection module attempt to detect a 

20 watermark in each of the blocks, and to determine its orientation. If one is identified in a 
block, the detector module invokes a message reader module to perform a read operation 
using the orientation parameters to align image data and extracting an auxiliary message 
embedded in the watermark. The specific details of the detection and read operations 
depend on the specifics of the watermark algorithm and watermark signal protocol. 
25 Some examples of these functions are set forth in detail in US Patent No. 5,862,260, and 
in co-pending application 09/503,881, filed February 14, 2000; which are hereby 
incorporated by reference. 

Fig. 2 is a diagram of an alternative streaming mode watermark decoder. Like the 
decoder in Fig. 1, this one operates on sequential bands 200 of an image from another 
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process or device. It buffers incoming bands in a band FIFO 202. The cache manager 
204 analyzes image blocks in the FIFO 202, selects candidate blocks for phantom 
caching in a block cache 206, and triggers a watermark detector module 208 using similar 
criteria as described previously. The watermark detector and message reader modules 
5 (208, 2 1 0) operate in a similar fashion as the decoder in Fig. 1 . 

This decoder in Fig. 2 differs from the one in Fig. 1 in that the FIFO buffers 
additional image data and the block cache does not store image data. Instead, the cache 
manager maintains a phantom cache in which it maintains information about the location 
of each block, but the not the corresponding image data for the block. When it triggers 
10 the detector, the cache manager provides the address of the phantom cached blocks in the 
;| FIFO (e.g., a pointer) to the detector module 208. The detector module then reads the 
J j blocks from the band FIFO directly, rather than reading them from the block cache as in 
i} the decoder shown in Fig. 1 . 

^ Fig. 3 is a diagram of a streaming mode encoder. The streaming mode encoder 

j Jl5 receives incoming, sequential bands 300 of an image. It buffers these bands in a band 
^ FIFO 302 that is at least one block in height. A block in the context of image watermark 
I I encoding refers to the size of image data into which a watermark encoder module embeds 
^ an entire watermark signal instance. The FIFO includes two separate buffers, enabling 
;j the embedder to load one with incoming data while performing embedding operations on 
*20 image blocks in the other one. 

When one of the band FIFOs is filled with new image data, a perceptual analyzer 
304 analyzes the pixel data in the block to compute a perceptual mask. In one 
implementation, this perceptual mask is an array of watermark gain control values used to 
modulate the strength of corresponding samples in an image watermark signal. Each gain 
25 control value corresponds to an element or group of neighboring elements in the spatial 
domain of a watermark signal. The gain control values are computed as a function of the 
corresponding samples in the host image being watermarked. In particular, they are a 
function of the local signal activity (e.g., local contrast and image signal edge 
measurements). For more information about computing a perceptual mask, see US Patent 
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Applications 09/596,698, filed on June 19, 2000, and entitled Perceptual Modeling of 
Media Signals Based on Local Contrast and Directional Edges; and 09/503,881, filed 
February 14, 2000. 

Separately, the watermark encoder prepares an auxiliary message 306 for 
5 encoding into the image. The message is comprised of a set of binary or M-ary symbols. 
In this implementation, the message is repeated in each block. However, it is possible to 
vary the message over different parts of an image by changing the message input to the 
watermark encoder. 

The message coding module 308 transforms the message into a watermark 
10 message signal. This process includes adding control bits to the message (fixed codes 
I that assist in aligning the image, and error detection symbols derived based on the other 
\ message symbols). It also includes error correction encoding, such as using convolution, 
BCH, Reed Solomon, turbo codes, etc. Finally, it includes a spread spectrum modulation 
= function in which the error corrected message symbols are spread over a pseudo random 
i 1 5 carrier image. In particular, the encoder spreads each binary symbol of the error 
? correction encoded message over a pseudorandom number using a spread spectrum 
i modulator (e.g., XOR for binary symbols, multiplier for binary antipodal symbols), and 
, the resulting binary signal elements are mapped to locations in the host image block. 
| A watermark embedding function 3 1 0 embeds the watermark message signal into 

20 the host signal, using the perceptual mask from the perceptual analyzer to modulate the 
signal intensity. There are a variety of watermark embedding functions detailed in 
watermarking literature, including patents and other publications. The embedding 
function may modulate spatial domain pixel values in a selected color channel (e.g., 
chrominance or luminance), or modulate frequency coefficients such as wavelet, Fourier, 
25 DCT transform coefficients, etc. The embedding function may modulate signal 

amplitudes or some other statistical feature such that that feature corresponds to a desired 
message signal element to be encoded. In one implementation, the embedding function 
modulates amplitudes of spatial pixel values in the luminance channel. Selected pixels 
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are adjusted up, while others are adjusted down corresponding to a corresponding 
element in the watermark message signal. 

In addition, in some applications where the watermarked signal is expected to 
survive geometric transformation, the watermark embedding function also embeds a 
5 watermark orientation signal (also referred to as a calibration or synchronization signal). 
This signal comprises an array of impulse functions that form peaks in the Fourier 
Magnitude domain of the image block. The impulse functions have pseudorandom phase 
to make them imperceptible or nearly imperceptible, yet their known phase enables the 
watermark detector module to compute the origin (e.g., the vertical and horizontal offset 
1 0 or translation) of the watermark signal in a distorted version of the watermarked image. 
I The detector module correlates the orientation signal with a log polar remapping (Fourier 
s Mellin transform) of the watermarked signal to determine rotation and scale parameters 
j of the watermark signal in a distorted version of the watermarked signal. The watermark, 
* including the orientation signal, is imperceptible or substantially imperceptible in the 
|15 watermarked signal. 

% The streaming mode encoder transfers watermarked blocks 312 into another FIFO 

I buffer 3 14. This FIFO is also double buffered, enabling the embedding function to fill 
, one FIFO, while transferring bands from the other buffer to the next stage in the printing 
{ or scanning process in which the streaming mode encoder is incorporated. 
20 The watermark systems depicted in Figs. 1-3 can be implemented for a variety of 

different applications and watermarking algorithms. This architecture is particularly well 
suited for incorporating watermark decoder functionality into a printer or scanner driver. 
In both cases, the driver is a software interface to a peripheral device, namely a scanner 
or printer, or a device that includes both image scanning and printing subsystems such as 
25 a fax machine or copier. The driver executes on a processor in a computer connected to 
the peripheral device. The driver enables application programs executing in the 
computer, including the operating system, to communicate with the peripheral. The 
application program communicates data and instructions to the driver via a programming 
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interface, sometimes referred to as an API. The driver communicates hardware specific 
instructions and data to and from the peripheral. 

Typically, these types of software drivers only pass sequential portions of an 
image to or from the peripheral during printing or scanning operations. As such, a 
streaming mode watermark architecture is particularly well suited for performing 
watermark functions on a sequential stream of image data as it is being transferred 
between the driver and peripheral. 

The streaming mode architecture is also well suited for use within printer or 
scanner devices, including multi-function devices like fax machines, copiers, and 
integrated copier, fax machine, printer and scanner devices. Many of these devices have 
internal processors and memory that manage the internal operations of the imaging 
device. For example, to manage printing operations, such devices include a combination 
of software, hardware, and firmware. 

In typical printing process, an image passes through a number of stages referred to 
as the printer pipeline. This pipeline may be implemented in a driver, in the printer, or in 
some combination of the driver or printer device. The printer pipeline typically includes 
a raster module and a render module. The input to the pipeline is either text or an image. 
Images are commonly input in a standard digital image format, such as array of RGB and 
YUV pixels. The raster module performs image processing functions like stretching, 
anti-aliasing to convert the image into a desired size and resolution. The render module 
then takes the rasterized image and coverts it into a binary bitmap (e.g., halftone format) 
for printing. For color printing, the render module performs a color mapping to the color 
format of the printer, such as a CMYK representation, where each color channel has a 
corresponding binary bit map or halftone representation. 

The streaming mode architecture is well suited for the printer pipeline within a 
printer device or driver because various stages of the pipeline operate on and pass 
sequential portions of the image to subsequent stages. To incorporate watermark 
functionality, a watermark module intercepts sequential portions of image data at a 
selected stage in the pipeline and applies the watermark function. 
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Within a printer, for example, a watermark function may be applied to sequential 
portions of the image as it enters the rasterizing stage, as it enters the rasterizing phase, or 
as it exits the rasterizing stage. 

The streaming mode architecture is also suited for scanner pipeline within a 
5 scanner device or driver. The scanner pipeline receives an image as it is being captured 
via an image sensor. The scanned image data typically gets color mapped into a standard 
form such as RGB or YUV and sent in sequential portions to another processing stage or 
device. A watermark function may be applied to the sequential portions of an image as it 
is captured from the image sensor or after it is color mapped. 
1 0 The incorporation of a streaming mode watermark encoder or decoder in printer 

;=! and scanner drivers has a number of advantages. One advantage is that it enables the 
jjj watermark functionality to operate in a manner that is independent of the application 
y 1 program that utilized the driver to send or receive image data from an imaging peripheral. 
!«=== There are a number of design issues relating to incorporating watermark functions 

! ! J 5 in drivers and imaging devices. One issue in drivers compatible with the Windows 
!= operating system is the software code format. Many drivers are written in 16 bit code. 
J Such drivers may require "thunking" across a 1 6 to 32 bit boundary in cases where the 
; = ' streaming mode watermark function is written in 32 bit format. 
□ One of the implementations of the invention includes 32 bit streaming mode 

=20 watermark function code integrated with a 1 6 bit printer driver compatible with the 

Windows 98 operating system. The printer driver calls an API of the watermark function 
code, which is implemented in C/C++ and compiled into a Dynamic Link Library that is 
linked to the printer driver at runtime. Thunking is a function call made across the 16:32 
bit code boundary. The printer driver code must make a thunk to pass incoming image 
25 bands of RGB data and related image data (such as image dimensions) to the watermark 
function API. The implementation minimizes the performance impact of this thunk by 
using a 16:32 bit bridge to the 32 bit watermark function code to pass the image band 
data. In particular, the bridge employs simple data structure types that include only band 
and image data necessary to perform the watermark function. The bridge includes 
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assembly code to ensure that the 32 bit code correctly addresses memory that is accessed 
using 16 bit pointers in the 16 bit driver. 

The implementation employs a thunk to pass image band data to the watermark 
function through its API. The API avoids a thunk back to the 16 bit code to return the 
results of the watermark function by filling in 16 bit parameters in the 16 bit code. These 
parameters represent the results of a watermark detection and read operation for a 
watermark decoder function. 

Another implementation issue is the image data model used to represent the 
sequential stream of image data. In a printer driver implementation for the Windows 98 
operating system, the image data model adheres to the data model established by the GDI 
module in the operating system. In a scanner driver implementation, the image data 
model adheres to the Twain Data Source standard for scanners. 

Another design issue that applies to implementations in drivers or devices is 
where to intercept the image data in the scanning and printing process. The printer driver 
implementation described above intercepts bands of RGB data from the driver before the 
rasterizing and rendering processes in the printer pipeline. This interception point 
enables the watermark code to hide its latency within the latency of the subsequent 
rasterizing and rendering process. As an alternative, the watermark function can also 
intercept RGB bands of image data after the rasterizing process but before rendering, or 
could intercept CMYK binary bit map data after the rendering process. 

Another design issue is whether to employ multi-threading in the streaming mode 
watermark function. Preferably the watermark function code (e.g., the watermark 
decoding and encoding functions) should execute on separate threads of execution so that 
the watermark function does not block the execution of other print or scanner pipeline 
processes. 

Another design issue is the extent to which the watermark function retains 
information from watermark operations on previous blocks. For example, the watermark 
decoder operation can be optimized by retaining and using results from previous 
operations. For example, in one implementation, the detector module detects the 
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watermark orientation signal by performing correlation (e.g., generalized match filtering) 
between the known orientation signal and the current image block in different transform 
domains, including the Fourier domain (magnitude or phase) and a log-polar resampling 
of the Fourier domain of the image block. Portions of an image block need not be 
transformed again into the Fourier domain or the log-polar resampling if they have been 
transformed in previous detection operations. Instead, the transformed data can be 
buffered in the detector module and re-used. 

Additionally, the message information decoded from different blocks can be 
compiled to determine the embedded message symbols more accurately. For example, 
rather than performing error correction decoding on spread spectrum demodulated 
information from a single block, raw message information (i.e. message information not 
yet error corrected) can be compiled from several image blocks, and then periodically 
error correction decoded as additional message information is collected from multiple 
image blocks. 

The geometric distortion, such as the scale, rotation, and translation, computed for 
previous blocks using the orientation signal is likely to be similar for other blocks. The 
orientation parameters for scale, rotation, translation, etc. can be used in subsequent 
correlation operations for additional image blocks. Rather than re-compute these 
parameters from scratch, the parameters for previous blocks can be used as starting points 
and refined in additional correlation operations. 

As noted previously, there are a number of applications of the streaming mode 
watermark encoder and decoder functions. The encoder and decoder can operate 
independently or together. For example, in response to detecting a watermark or specific 
watermark message information with the decoder, the encoder can then be invoked to 
encode a watermark in the image being scanned or printed in response to the results of a 
streaming mode decoder operating on the image during the scan or print operation. 

The watermark encoder can be used to embed tracer data in an image as it is being 
printed or transferred. The forensic tracer data may include: data identifying the date of 
an activity from a clock in the imaging device or host computer of the driver, data 
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identifying the serial number of a computer system, data identifying a serial number of a 
system component, data identifying a user of the computer system, data identifying a file, 
data indicating the nature of a detected event, data indicating the status of the computer 
system, data from a registry database, data relating to an external network connection, 
5 and data derived from a digital watermark payload. 

The encoder may be designed to embed this tracer data in the image in response 
to results of a streaming mode watermark decoding operation on the image. 

The watermark decoder may be used to initiate one or more counterfeit deterrence 
actions such as: stopping printing or scanning, spoiling the printing or scanning output 
10 image by introducing artifacts or a visible marking like ("void", or "copy") into the 
I image output, providing a notice to the user via a graphical user interface or audio output, 
j etc. Preferably, the streaming mode decoder initiates one or more of these actions before 
jj the printing or scanning operation has completed. Such an action can be triggered when 
* the watermark decoder detects a predetermined event. One event is detection that the 
1 1 5 correlation between the known watermark orientation signal and the watermarked image 

exceeds a threshold. Another event is accurately decoding a watermark message as 
1 determined by checking error detection symbols relative to other symbols in the 
watermark message. 

{ The watermark decoder may also be used to initiate actions on metadata 

20 associated with the image via the watermark embedded in it. Because these actions are 
triggered from a streaming mode decoder, they can be performed during the printing or 
scanning of the image. The metadata may be included in the watermark message 
payload, or referenced via an identifier in the watermark message payload. The decoder 
may trigger the rendering of the metadata, which includes presenting graphical, image, 
25 video or audio data on a user interface or executing program instructions in the metadata. 

The metadata may be stored in a database in the imaging device or in another 
device accessible to the imaging device or system via a wire or wireless network 
connection (wireless phone network, Internet, LAN, etc.). The watermark message may 
include an address, index, or URL. The decoder may trigger a programmatic process to 
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fetch related information or program instructions from that address, index, or URL, For 
example, the decoder may fetch a web page stored at the URL or provide the URL to 
another application program, such as an Internet browser, for fetching and displaying a 
web page at the URL. Alternatively, the decoder may send the index to a database, 
5 which in turn, provides corresponding information or instructions back to the decoder. 
The database record matching the index may include yet another reference to information 
or instructions, such as a URL to a related web site. The database (e.g., web server) may 
either return this information to the decoder or route it to another device (e.g., web 
server), which in turn returns related information or instructions to the decoder (e.g., 
10 computer or imaging device where streaming mode decoder is executing). For related 
^ information on such applications for using watermarks to link watermarked content to 
H information or actions, see US Patent No. 5,841,978 and US application nos. 09/571,422; 
jl 09/563,664; and 09/574,726. 

ll The watermark encoder may be implemented to include a user interface to enable 

Jil5 the user to control whether an image is watermarked while being printed or scanned, and 

to specify the information to be embedded in the watermark. For example, the watermark 
j I encoding functionality may be selected via the printer or scanner dialog box of the printer 

or driver software. In this dialog box, the user can enter alphanumeric text for encoding 
□ in the image as it is being scanned or printed. For example, the user can enter a reference 
*20 to an electronic version of the image being printed or scanned before it is printed or 
scanned. Additionally, the user can enter a reference to other information, such as a 
database record^ a URL, a program, or other related information or data. The user can, 
for example, insert a reference to a licensing web page, or usage control information. A 
compatible decoder can then automatically trigger actions, such as rendering the metadata 
25 associated with the watermark message during subsequent printing or scanning 
operations on the image. 

The watermark decoder may also have a user interface to enable the user to 
control watermark decoding, to apprise the user of decoding results, and to render 
metadata (information or programs) associated with an image via a watermark. 
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The watermark decoder can be further enhanced by combining it with additional 
modules for detecting visible patterns used in counterfeit deterrence, such as visible 
patterns of shapes used on bank notes. The pattern recognition module or modules can be 
spawned on separate threads yet operate on the same image data in the FIFO buffer as the 
5 watermark decoder. In the event that the pattern recognition module detects a 

predetermined pattern of a bank note or other secure document it triggers an action, such 
as inhibiting the print or scan process before it completes, informing the user, linking to a 
database or web site, etc. For more information on pattern recognition modules, see co- 
pending application 60/176,693, filed January 18, 2000, and entitled Multistage Detection 
1 0 of Geometrical Structures or Patterns, which is incorporated by reference. 

Concluding Remarks 

= j Having described and illustrated the principles of the technology with reference to 

|| specific implementations, it will be recognized that the technology can be implemented in 
1 5 many other, different, forms. To provide a comprehensive disclosure without unduly 
: j lengthening the specification, applicants incorporate by reference the patents and patent 
=i applications referenced above. 

=1 The methods, processes, and systems described above may be implemented in 

:i hardware, software or a combination of hardware and software. For example, the 
20 watermark encoding processes may be implemented in a programmable computer or a 
special purpose digital circuit. Similarly, watermark decoding may be implemented in 
software, firmware, hardware, or combinations of software, firmware and hardware. The 
methods and processes described above may be implemented in programs executed from 
a system's memory (a computer readable medium, such as an electronic, optical or 
25 magnetic storage device). 

The particular combinations of elements and features in the above-detailed 
embodiments are exemplary only; the interchanging and substitution of these teachings 
with other teachings in this and the incorporated-by-reference patents/applications are 
also contemplated. 
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We claim; 

1 . A method of image watermark decoding in a printing or scanning process 
comprising: 

intercepting portions of an image as the portions pass from one stage of the 

printing or scanning process to another; 

performing a watermark decoding operation on each portion; and 
providing a result of the decoding operation before the printing or scanning 

process has completed for the image. 

2. The method of claim 1 wherein the image portions are sequential image 
portions, and all decoding operations on one portion are initiated before any watermark 
decoding operations are initiated on a subsequent portion in a sequential data stream of 
the sequential image portions. 

3. The method of claim 1 wherein the watermark decoding operation is 
performed in a printer driver executing in a computer as an image is being passed from an 
application program to a printer through the driver. 

4. The method of claim 3 wherein the printer driver includes 16 bit code, the 
watermark operation is implemented in 32 bit code, and the watermark operation is 
invoked from the 16 bit code through an application programming interface of the 32 bit 
code. 

5. The method of claim 4 wherein the 16 bit code passes image data to the 32 bit 
code over a 16 to 32 bit bridge, and the bridge includes code enabling the 32 bit code to 
access data structures in the 16 bit code. 

6. The method of claim 1 wherein the portions are buffered, and analyzed to 
select blocks for watermark detection operations. 
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7. The method of claim 6 wherein the analysis of the blocks in the buffer includes 
identifying potentially overlapping blocks that are likely to include a watermark signal 

5 8. The method of claim 1 wherein the result of the decoding operation is used to 

trigger an action before printing or scanning of the image is complete. 

9. The method of claim 8 wherein the action includes stopping the printing or 
scanning of the image. 

10. The method of claim 8 wherein the action includes using information in the 
watermark to index related information about the image in a database. 

1 1 . The method of claim 8 wherein the action includes using information in the 
watermark to fetch a web page related to the image. 

12. A computer readable medium on which is stored software for performing the 
method of claim 1. 

^20 13. An imaging system comprising: 

an imaging device for scanning or printing an image; 

a watermark decoder in communication with the imaging device for intercepting 
portions of an image as the portions pass from one stage of a printing or scanning process 
to another, for performing a watermark decoding operation on each portion; and for 
25 providing a result of the decoding operation before the printing or scanning process has 
completed. 
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14. The system of claim 13 including a printer peripheral in communication with 
a computer, and a printer driver executing in the computer and incorporating the 
watermark decoder. 

15. The system of claim 13 including a scanner peripheral in communication with 
a computer, and a scanner driver executing in the computer and incorporating the 
watermark decoder. 

16. A method of image watermark encoding in a printing process comprising: 
intercepting portions of an image as the portions pass from one stage of a printing 

process to another; 

performing a watermark encoding operation on each portion; and 

providing watermarked portions of the image to a subsequent stage in the printing 

process. 

17. The method of claim 16 including encoding tracer data into the image. 

18. The method of claim 17 including encoding tracer data into the image in 
response to detecting a watermark in the image. 

19. The method of claim 16 including encoding calibration data into the image 
that is operable to detect a watermark in a geometrically distorted version of the 
watermarked image. 

20. A computer readable medium having software for performing the method of 
claim 16. 
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Watermark Encoding and Decoding in Imaging Devices and 

Imaging Device Interfaces 

Abstract of the Disclosure 

The disclosure details methods and systems for watermark encoding and decoding 
5 in imaging devices, such as printers and scanners, and imaging device interfaces. 

Watermark encoding and decoding functions are incorporated into printer and scanner 
image processing pipelines. Watermark encoders and decoders perform watermark 
encoding and decoding, respectively, on an image as it is being transferred from one 
stage of a printing or scanning process to another. In particular, streaming mode 
10 watermark encoders and decoders operate on sequential portions of the image in a 

sequential image stream passing from one stage to another. The streaming mode encoder 
and decoder may be incorporated in a printer or scanner driver in a computer connected 
to a printer or scanner peripheral or within a printer or scanner device. 
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COMBINED DECLARATION AND POWER OF ATTORNEY 
FOR PATENT APPLICATION 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name, 
I believe I am the original, first and sole inventor (if only one name is listed below) or an 
original, first and joint inventor (if plural names are listed below) of the subject matter which is 
claimed and for which a patent is sought on the invention entitled WATERMARK ENCODING 
AND DECODING IN IMAGING DEVICES AND IMAGING DEVICE INTERFACES, the 
specification of which 

[x] is attached hereto. 

[ ] was filed on as Application No. . 

[ ] was described and claimed in PCT International Application 

No. , filed on , and as amended 

under PCT Article 19 on (if applicable). 

[ ] and was amended on (if applicable). 

[ ] with amendments through (if applicable). 

I hereby state that I have reviewed and understand the contents of the above-identified 
specification, including the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose information which is material to patentability as defined in Title 
37, Code of Federal Regulations, § 1 .56. If this is a continuation-in-part application filed under the 
conditions specified in 35 U.S.C. § 120 which discloses and claims subject matter in addition to that 
disclosed in the prior copending application, I further acknowledge the duty to disclose material 
information as defined in 37 CFR § 1 .56 which occurred between the filing date of the prior application and 
the national or PCT international filing date of the continuation-in-part application. 

I hereby claim foreign priority benefits under Title 35, United States Code, § 1 19(a)-(d) of any 
foreign application(s) for patent or inventor's certificate or of any PCT International application(s) 
designating at least one country other than the United States of America listed below and have 
also identified below any foreign application(s) for patent or inventor's certificate or any PCT International 
application(s) designating at least one country other than the United States of America filed by me on the 
same subject matter having a filing date before that of the application(s) on which priority is claimed: 

Prior Foreign Application(s) Priority 

Claimed 

[X] [] 

(Number) (Country) (Day/Month/Year Filed) Yes No 

I hereby claim the benefit under Title 35, United States Code, § 1 19(e) of any United States 
provisional application(s) listed below: 


Application Number Filing Date 
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I hereby claim the benefit under Title 35, United States Code, § 120 of any United States 
application(s) or § 365(c) of any PCT International application(s) designating the United States, listed 
below and, insofar as the subject matter of each of the claims of this application is not disclosed in the 
prior United States or PCT International application in the manner provided by the first paragraph of Title 
35, United States Code, § 1 12, 1 acknowledge the duty to disclose material information as defined in Title 
37, Code of Federal Regulations, § 1.56(a) which occurred between the filing date of the prior application 
and the national or PCT International filing date of this application: 


(Application No.) (Filing Date) (Status: patented, 

Pending, abandoned) 

The undersigned hereby authorizes the U.S. attorney or agent named herein to accept and follow 

instructions from as to any action to be taken in the Patent and Trademark Office regarding 

this application without direct communication between the U.S. attorney or agent and the undersigned. In 
the event of a change in the persons from whom instructions may be taken, the U.S. attorney or agent 
named herein will be so notified by the undersigned. 

I hereby appoint the following attorney(s) and/or agent(s) to prosecute this application, to file a 
corresponding international application, and to transact all business in the Patent and Trademark Office 
connected therewith: 


William Y. Conwell 
Joel R. Meyer 
Thomas M. Horgan 
Elmer Galbi 


Reg. No. 31,943 
Reg. No. 37,677 
Reg. No. 33,183 
Reg. No. 19,761 


Address all telephone calls to Joel R. Meyer at telephone number (503) 968-0443. 
Address all correspondence to: 


Joel R. Meyer 

DIGIMARC Corporation 

Digimarc Corporation 

19801 SW 72nd Avenue, Suite 250 

Tualatin, OR 97062 


I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements were 
made with the knowledge that willful false statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1 001 of Title 1 8 of the United States Code and that such willful false 
statements may jeopardize the validity of the application or any patent issued thereon. 


Full Name of Sole or first Inventor: Clayton L Davidson 


Inventor's Signature 

Date 

Residence: Lake Oswego, Oregon 
Citizenship: USA 

Post Office Address: 17374 Lake Haven Drive, Lake Oswego, OR 97035 
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Full Name of Second Joint Inventor: Aruna B. Kuman 
Inventor's Signature 

Date 

Residence: Camas, Washington 
Citizenship: India 

Post Office Address: 2253 NW 22 nd Cir, Camas, WA 98607 
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